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To observe that post-'secondary education is now being overrun with techniques 
^1 for student evaluation of courses, via questionnaires, checklists and sundry 



other devices^ is, by now, trite. It seeros like every campus has its o\m spe- 
cial form and many have several forBs, One night hope that from these efforts 
would spring a Qood deal of variety. This, hox^^ever, seems clearly not to be 
the case. If one were to line up side by side the five or six most frequently 
cited instruments, meaningful differences between the instruments would be hard 
to detect. 

Probably the most characteristic feature of all these instruments is that 
they all focus attention on the course — what is going on — and the instructor 
— - what is he or she doing. The central focus of the educational enterprise — 
the student — is lost sight of. From some perspectives ^ focusing attention on 
. the course and instructor nay be legitimate. On the other hand, focusing atten- 
tion on the student vrould also seeni to be a lecitirnate perspective for approach- 
ing the question of student evaluation of courses. The basic question becomes^ 
IJliat has been the impact of the course on the student, in the student *s owa 
judgment? The purpose of this paper is to explore this approach to student 
evaluation of courses by examining some concrete attempts to. use this approach, , 

qo: ■ 

TWO IIJSTRUMEHTS 

We describe here two instruments developed at the Unlver^i^ty of V71f3 cons In- 
Green Bay within the past four years which focus on aspects of student develop-- 
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ment. The first of these instruments is knotm as the Course Comments Question-- 
naire (CCQ) . The CCQ consists of 35 Likert-type items , yielding scores on 
seven different, factors . The factors, as described in the Instructor's Manual 
(Hogan, ]973b), and their associated items are displayed in Table 1. 

These factors are an outgrowth of factor analytic work originally reported 
by Hartley and Ilogan (1072). Other work using this instrument but not directly 
relevant to the present discussion may be found in Hogan (1973a) and Kulik & 
Kulik (1974). 

The first four factors in the CCQ correspond closely to those identified 
in many other factor analytic studies which have employed traditional courpti/ 
instructors-oriented questionnaires. These factors deal with ins tructor-^student 
relationships 5 organization^ the difficulty level of the course, and an overall 
judgment about both the course and instructor. 

The last three factors represent the unique contribution of the CCQ in the 
area of student evaluation. These factors are, as simple inspection of the 
items in Table 1 makes evident, learner-oriented. The first of these factors 
deals XTith what we have called '^general cognitive development j" the development 
of cognitive abilities such as reasoning and working creatively without specific 
reference to the field being studied in the course. (Item 22, although turning 
up on this factor, is not consistent with the interpretation of the factoi,, 
for no readily apparent reason.) 

The second learners-oriented factor has been labeled ' specific cognitive 
development.'^ It deals with students' self^-perceived progress in knowledge ^ 
skills, etc. directly related to course content. 

The third learner-oriented factor deals with the self-perceived development 
of interests J appreciations 3 concernSp and personal applications. 

As suggested by the factor analytic origin of the CCQ scales j. the learner- 
oriented items tap dimensions of student reactions to courses other than such 
traditional dimensions as course organization and instructor responsiveness . 



Subsequent factor analyses which we have conducted with the CCQ items have 
confirmed the separateness of learner-oriented and course/instructor-oriented 
dimensions although subdivisions within these broad areas have not always 
duplicated those originally identified. The CCQ presently includes three such 
learner-oriented diinensions but it seems unlikely thai: these three exhaust the 
possibilities in this area. Hopefully^ additional research will uncover addi- 
tional learner-oriented dimensions which will further enrich our perspectives 
on student reactions. 

A second instrument which we have developed extends the learner-orientation 
even farther than that represented in the CCQ. This instrument kno^m as the 
Personalized Course Ajialysis (PGA) 5 consists of twenty goal statements. These 
statements, shown in Table 2, were designed to reflect the varieties of goals 
which students might see as important for their own purposes. 

Our analysis of possible goals was based on reviews of statements such as 
those by WICI-IE (e.g. Lawrence^ Weathersby^ and Pattersonj 1970, Hicekj Services 
and Lee, 1974) ^ those contained in the Institutional Goals Inventory (1971), 
and similar documents, as xvell as our o^vn, local emphases. 

Two of the PCA goal areas, General Cognitive Developnient and Specific 
Content, correspond to scales covered in the CCQ. In addition, the PCA covers 
Job-related goals. Cultural emphases, Social/Political emphases. Personal 
Development, and Communication skills. 

In the usual mode of administration for the PCA, a student makes two res- 
ponses to each goal statement (each rating on a scale from 1-10). First ^ the 
student indicates how important the goal is for him: second, the student indi- 
cates how much progress he feels he has made. With the PCA a student may indi- 
cate that he made little progress in terms of developing job--related skills but 
that this kind of goal really had very little importance for him; or that he 
felt . he made some progress in his personal development but, since this; was the 
most impprtarit goal area for himv just ''some ' progress was a disappointing result 
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Factor analysis of student responses to the goals strictly in terms of the 
'importance-' ratings suggests that the Specific Content and Career/Job goals 
mainly define a single dimension and that Cultural Emphasis and Social/Political 
Emphasis may define one dimension. Personal Development seems to stand by it- 
self and General Cognitive Development separates out nicely. It is not clear 
how the Communication items fit into the picture: the two items presently 
placed in this category do not have good communality even on a priori grounds. 
In any case^ we intend to undertake more extensive factor analytic work with 
the PCA instrument before attempting to revise it. 

After initial use of the PCA in its intended- mode of administration, it 
quickly became apparent that delaying the inquiry regarding importance of goals 
till the end of the course was rather silly? this inquiry should be made at the 
beginning of the course. Therefore, a Personalized Prej-Course Analysis was dev- 
eloped which involves use of the PCA statements with students marking only the 
importance ratings. Pvesults are typically returned to tha instructor within 
the first two weeks of class. 

Some of our most interesting results come from review of data from the 
Pre-Course Analysis. Four of these findings will be mentioned here. None of 
the results are particularly startling: thay could probably be predicted in a 
rough fashion by many experienced faculty. The principal value of the PCA 
lies in documenting trends in a systematic way and in identifying exceptions to 
general trends. 

First, we have noted considerable variation in the pattern of goal impor- 
tance within curricular areas as well as across curricular areas. Figure 1 
shows average class ratings for the seven areas covered in the PCA (used at the 
beginning of the semester) for three courses. The interesting point here is 
that all of these courses are in the same area (psychology). On the average , 
students in these three courses are looking for rather different things. Even / 
•larger differences can be noted when reviewing results from courses in different 



curricular areas. 

Second a the PCA reveals noteworthy differences in the goals of indivi- 
dual students within the sade course. Figure 2 shows the goal patterns for 
four students taken from course *^A'' in Figure 1. Without even knowing what 
course '^A'' is all about ^ it is interesting to speculate how the instructor 
might differentiate content, assignments, etc, for these students j particularly 
for student #2, This kind of informal analysis immediately suggests the need for 
the more systematic kinds of analyses provided by clustering techniques. This 
is one of the areas in which we are attempting to make progress at the present 
time. 

Thirdly, in a somewhat different vein, there is one goal area that seems 
to receive uniformly high ratings across nearly all students and courses. 
This is area #1; general cognitive development. Students in upper division 
and loxi/er division courses, in humanities, social sciences, and natural sciences, 
all seem to prize this area. 

Finally, results of an entirely separate part of our evaluation v7ork are 
beginning to converge with our efforts in the area of personalized course ^val- 
uation. We have recently undertaken a comprehensive assessment of "the educa- 
tional goals and previous experiences of our incoming freshmen. Factor analyses 
of these data have turned up almost exactly the same factors as those represented 
in the PCA (Hartley, 1976). To some extent this sexryes as evidence for the 
construct validity of the PCA, More importantly j/it suggests the possibility 
of analyzing student goals^ and progress toward those goals^^both in terms of 
total educational experience and particular educational experiences in a con- 
joint fashion, 

50ME ADVMTAGES AITO DISADVANTAGES 

' In the discussion of instruments and results above, some of the advan- 
tages and disadvantages of the GCQ and PCA types of instruments may have been . 
hinted: at. An explicit discussion of these advantages and. disadvantages may ^ .^^ 



be helpful, especially to individuals who may be contemplating use of such 
instruments on their own campuses. 

The major advantage of the types of ins3trunents treated here, obviously, 
is that they focus attention on the student rather than on the instructor or . 
instructional process. They serve as a constant reminder to both student and 
instructor that the central concern In the educational enterprise is the student 
and what happens to the student. 

Additional advantages of learner^-oriented course evaluations are outgrowths 
of 'the latter 3 general point of view. Re find that these instruments apply to 
virtually any type of educational endeavor. With the more traditional types of 
questionnaires 3 requests are constantly received for separate instruments or 
at least some separate questions^ for team-taught courses, lab sections, discus- 
sions-based courses, field experiences , etc. The learners-oriented questionnaire, 
on the other hand, seems to apply about equally well to any kind of course or 
any kind of educational experience. 

Another advantage of the learner-oriented end~of -course evaluation is that 
student responses may be aggregated across courses and related to institutional 
goals. Many institutions have goals in such areas as appreciation of the arts, 
development of social responsibility ; preparation for careers, and so on. 
The extent to which students see themselves as progressing in these areas may 
be assessed by questionnaires given to seniors or graduates. But this is 
usually difficult to do, requiring as it does a major research undertaking. 
However, if learner-oriented course evaluations are used in many courses, it 
becomes possible to provide at least some information about self-perceived 
progress of students in such areas' by simply summing data across courses. In ^^^-^ 
contrast, relating averages on the traditional types of questionnaires Jto'^insti- 
tutional goals is not very meaningful. For example, in terms of ^institutional : 
■ goals, , who really caires if most courses at a school are per.ce'lved as well-organ-- 
Ized , : or ! very^d or suffused with good student^ns true tor relationships? ; 
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A final advantage of learner-oriented instruments. espGcielly of the PCA 
type, is that they allow an instructor to identify student concerns at the ■ 
beginning of a course and plan instruction accordingly or, at least, proceed 
with the kn^ ;ledge that thare is some discrepancy betT/een the instructor's 
goals and the students' goals for a course. 

Of course, things are not all rosy. There are some disadvantages encoun- 
tered in the use of learner-oriented course evaluations. Perhaps foremost 
among the difficulties is faculty attachment to the traditional kind of question- 
naire. Many instructors seem to be more interested in whether students thought 
the course was well organized than whether the students thought they learned 
anything: or more interested in student reactions to quizzes than in v/hether the 
course met students' needs. This difficulty is compounded when a-particular 
rating dimension does not have an underlying '*good-bad" connotation. Individual 
instructors and faculty reviexcr committees have become accustomed to looking at 
certain types of information and interpreting it in certain ways. Substituting 
for the customary information a new type, oriented toward the learner rather 
than the instructor^ is not accepted with universal enthusiasm, We have not yet 
developed a satisfactory solution for this problem. 
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Table 1. List of Factors and Items in Course Comments Questionnaire (CCQ)^ 



I- GLOBAL RATING 

This scale or dimension appears to be the students' overall, suraraative 
judgment about the course. It is based on the following items: 

Did the instructor put the material across in an interesting way? 
Would you take another course from this instructor if you didn't have to? 
Considering everything, how would you rate the teaching in this course? 
Would you recommend this course to a friend who didn't have to take it? 
Considering everything, how would you rate this course? 

.II. RESPONSIVENESS 

This scale deals with the relationship between the instructor and students. 
Another name for the scale might be "rapport." Previously, this scale was called 
"Interaction" but it was decided that "Responsiveness" xaore accurately described 
.the dimension. This scale is b'ased on the following items: 

7 - Was the instructor actively helpful when students had difficulty? 

14 - Did the instructor appear sensitive to the students' feelings? 

21 - Was the instructor fair in his dealings with students? ^ 

28 - Were students free to ask questions, disagree, express their ideas, etc.? 

35 - Did the instructor tell students when they had done particularly well? 

Ill; DIFFICUL.TY 

This scale deals with how hard, difficult, or demanding the course was. On 
this scale a course might be rated as tod hard (high score) or too easy (low score) . 
This scale is based on . the following items: 



4 - Did the instructor make the students develop intellectual discipline? 
11 - Did students have to work hard tc meet course requirements? 
18 - Were students required to conform to high intellectual standards? 
25 - Was the amount of work required appropriate for the credit received? 
32 - Were the assigned readings and/or problems at an appropriate level? 

IV. ORGAl^IZATION 

This scale deals with matters of organization, preparation, and clarity of 
procedures as perceived by students. The scale is based on the following items: 



6. r, Did .the instructor use enough examples to clarify the material? . 
13 - Did /the instructor present material in a vyell organized fashion? 
20 - Were the methods used for evaluating (tests, grading projects, etc.) 
reasonable? 

27 Were the objectives of the course clear? 

34 — Were standards for grading clearly communicated to you? 



2 - 
9 - 
16 ^ 
23 - 
30 - 



Table 1. continued 



V. GENERAL COGNITIVE DEVELOPMENT 

This scale deals with development of cognitive abilities which are not 
tied directly to the content of the course. It is based on the following items: 

1 - I developed my ability to identify main points or central issues. 

8 - It is now easier for me to identify factors related to conclusions . 
15 ^ I develope'd- my ability to combine arguments and draw conclusions. 
22-1 developed significant skills in the field. 
29-1 developed my ability to function creatively, 

VI. SPECIFIC COGNITIVE DEVELOPMENT 

This scale deals with development in the cognitive area which is based 
rather directly on the specific course content. The scale is based on the 
following items: 

3-1 can recognize when people are using faulty arguments in this field. 
10 - In conversations, I can recall important information in this field. 
17 - I can now understand relatively advanced presentations on the subject. 
24 - I can identify values that enter into making judgments in this field. 
31 I became able to analyze new and complicated material in the field. 

VII. RELEVANCE 

This scale deals generally with the effect of the course on development of 
interest, concern, appreciation, etc. The scale is based on the following items: 

5-1 developed increased interest in the field. 
12-1 had discussions of related topics outside of class, 
19-1 became aware of ways t\\e subject is involved in my o\m life. 
26 - I increased my concern for community projects related to the course. 
33-1 appreciate things I didn't appreciate before. 



^From Hogan, T. P., UWGB Course Comments Questionnaire, Instructor's Manual. 
Office for Educational Development, University of Wisconsin-Green Bay, 1973. 
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Table 2. Major Goal Areas and Specific Items in Personalized Course Analysis (PCA) 



General Cognitive Development 
Item 

1 Improve ability to think, reason critically 

8 Learn to carefully analyze new materials, problems 
15 Develop problem solving or creative ability 

Career/Job Goals 

Item 

2 Attain skills or knowledge for a career or job 

9 Prepare for graduate or other advanced education 
.16 Increase my chances for getting a desirable job 

Cultural Emphasis 

Item 

3 Become aware of various cultures, ways of life 

10 Increase appreciation of cultural expressions 

17 Broaden outlookj develop new perspectives 

Specific Content 
Item 

4 Learn a particular body of knowledge 

11 Explore a variety of topics in this field 

18 Develop ability to apply skills in this field 

Social/Political .Emphasis 
Item 

5 Increase sensitivity to social problems 

12 Become involved in social and political concerns 

19 Learn to work more effectively as a citizen 

Personal Development 
Item 

6 Clarify my own , values , concerns, goals 

13 Increase self-understanding 

20 Increase self-reliance , personal maturity 

Communication (Verbal, Interpersonal) 
Item 

7 Improve communication skills, e.g. writing, speaking 

14 Learn to work more effectively with other people 



Fij»ure 1. Illustrative differences among^classes in terms of 

average importance of-:&t-udents ' self-expressed goals 
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Figure 2. Illustrative differences among students (from Course A, Fig. 1) 
in terms of importance of goals 
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